Skip to content

feat(snapshot): attach a fallback screenshot to sparse captures - #1764

Merged
thymikee merged 1 commit into
mainfrom
claude/sparse-a11y-tree-fallback-1eaa3e
Aug 16, 2026
Merged

feat(snapshot): attach a fallback screenshot to sparse captures#1764
thymikee merged 1 commit into
mainfrom
claude/sparse-a11y-tree-fallback-1eaa3e

Conversation

@thymikee

@thymikee thymikee commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

A user-facing sparse snapshot now captures the screenshot its recovery guidance already requires and returns the client-visible path as fallbackScreenshotPath. The image uses the normal artifact channel, so local and remote clients receive a usable path; internal observations and healthy snapshots do not trigger screenshots.

Only a sparse-tree verdict adds the app accessibility warning. Capture-limit verdicts remain attributed to the tool. The public contract, SDK normalization, digest projection, text output, architecture context, and snapshot documentation all describe the same response.

Scope: 16 files in the snapshot response/public projection. No command-family expansion. The agent-device skill remains unchanged because command behavior belongs in versioned help and product documentation.

Validation

  • pnpm check:affected --run — passed all 7 selected gates; 263 test files and 2,240 tests passed.
  • Production-route regression covers sparse screenshot capture, artifact publication/materialization, best-effort failure, and internal-observation exclusion.
  • Red-before proof: disabling the sparse fallback made the production-route test fail with expected 1 screenshot call, received 0.
  • Live iOS simulator: source-built daemon opened Settings, returned a healthy tree without fallbackScreenshotPath, and closed the isolated session cleanly.

Residual: the repository has no live app fixture that deliberately publishes a sparse accessibility tree, so the sparse-device branch is proven through the production-route regression rather than a live sparse capture.

@github-actions

github-actions Bot commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 2.26 MB 2.26 MB +1.6 kB
JS gzip 743.4 kB 743.9 kB +498 B
npm tarball 861.1 kB 861.6 kB +544 B
npm unpacked 3.00 MB 3.01 MB +1.9 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 29.8 ms 27.0 ms -2.8 ms
CLI --help 65.8 ms 65.8 ms -0.0 ms

Top changed chunks:

Chunk Raw diff Gzip diff
dist/src/runtime.js +265 B +105 B
dist/src/internal/daemon.js +267 B +70 B
dist/src/prepare-kind.js +259 B +70 B
dist/src/agent-device-client.js +98 B +20 B
dist/src/session2.js 0 B -3 B

@github-actions

github-actions Bot commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-08-16 14:06 UTC

@thymikee

thymikee commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

Reviewed exact head 63a1c078. The fallback screenshot dispatch structure is sound: it sits on the public snapshot response seam, skips observationOnly polling captures, composes its warning, and cannot fail the requested snapshot.

P1 — sparse-tree is not proof of an app accessibility defect. The runner stores the first capture-plan failure in the terminal verdict. A tree backend can set firstFailure to sparse-tree, while later query/private-AX tiers fail, exhaust budget, are unavailable, or never run; the final sparse payload still carries that first reason code. Therefore the new warning can falsely claim every backend reached the screen and that assistive technologies see an empty tree. It also contradicts the adjacent CONTEXT invariant that iOS sparse/AX failures are not proof of empty UI. Remove this attribution, or introduce an explicit aggregate terminal fact proving every applicable independent semantic backend returned semantic emptiness. Add a mixed-outcome regression showing tree-sparse plus later failure/budget/unavailable does not blame the app.

Before readiness, provide exact-head live iOS evidence on a genuinely sparse screen that the linked PNG exists and is readable, while internal polling does not capture repeatedly; the current mocked test proves dispatch decisions but not a usable artifact. Remove the unrequested skills/agent-device/SKILL.md behavior edit per the repository hard rule—versioned CLI help owns command behavior guidance. Finally rerun iOS Smoke to green: the current automation-longpress visibility failure is a known unrelated flake, but red CI still gates readiness. No readiness label applied.

@thymikee
feat(snapshot): attach a fallback screenshot to sparse captures
c42e164 
A sparse verdict already tells the caller to use a screenshot as visual truth,
which made that screenshot the guaranteed next command on every unreadable
screen — a second round trip to obey advice we authored. The user-facing
`snapshot` dispatch now takes the shot itself and links the path in its
warnings.

The fallback is deliberately hung off `dispatchSnapshotViaRuntime` and skipped
for internal observations: selector resolution, settle, and wait polling reach
`captureSnapshot` directly, so a wait polling an unreadable screen cannot turn
into a screenshot per poll. A failed shot is swallowed — the verdict's own
warning still carries the manual remedy, so the fallback can never fail the
snapshot that was asked for.

Sparse captures also say when the screen is the app's problem. Only the
`sparse-tree` reason code is evidence about the app: every backend reached the
screen and it published no semantic content, which is the same emptiness
assistive tech gets. `ax-rejected`, `budget`, `no-nodes` and `capture-failed`
are limits of this tool and stay unattributed, so readers are not sent to file
bugs against code that is not broken.
@thymikee
thymikee force-pushed the claude/sparse-a11y-tree-fallback-1eaa3e branch from 63a1c07 to c42e164 Compare August 16, 2026 13:38
@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Aug 16, 2026
@thymikee

thymikee commented Aug 16, 2026

Copy link
Copy Markdown
Member Author

Rebased onto current main and reviewed the final head c42e1642f. Code review is clean: the sparse screenshot now uses the normal artifact/materialization path, healthy and internal snapshots stay side-effect free, and app-defect wording is limited to sparse-tree. Local affected validation passed all 7 gates (263 files / 2,240 tests), including recording-artifact regressions; live iOS healthy-path verification also passed. Marking ready for human review while the new-head GitHub checks complete. Residual: there is no live sparse-tree fixture, so that branch is covered by the production-route regression and red-before proof.

@thymikee
thymikee merged commit f378050 into main Aug 16, 2026
33 checks passed
@thymikee
thymikee deleted the claude/sparse-a11y-tree-fallback-1eaa3e branch August 16, 2026 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@thymikee